Biostats Lecture 5: Statistical Hypothesis Testing

class: center, top, .title-slide, title-slide

# Biostats Lecture 5: Statistical Hypothesis Testing
## Public Health 783
### Ralph Trane
### University of Wisconsin–Madison 
### Fall 2019

---

# Recap

Random variables

Distributions

Estimators

Estimators are random variables! (for example, the average is a random variable)

---
layout: true

# Statistical Hypothesis Testing

---

**Scenario**:

We've been playing a simple game. Everytime you roll a six, I pay you a dollar. Everytime I roll a six, you pay me a dollar.

I've had crazy good luck, and by the end of the day won a lot of money from you.

You accuse me of cheating, and demand to test the dice I've been using!

I agree to let you test them, but ONLY if you do it in a sound, statistical manner. How to go about that?

You decide to roll the dice `$8$` times each, for a total of `$42$` rolls. You assume they'll all behave the same, so the probability of rolling a six is the same for all three dice.

---

**Setup**:

Let `$X_1, X_2, ..., X_{42}$` be the outcomes of the thirty "trials". Each trial consists of rolling a die, and check if it's a six or not. If it's a six, we'll call it a success, if not we'll call it a failure. I.e. `$X_i \sim$`
--
 `$\text{Bernoulli}(\pi)$`.

*IF* the dice are fair, `$P(X_i = 1) = 1/6$` for all `$i = 1,2,...,42$`.

*IF* the dice are fair, we would expect to roll a `$6$` close to `$\frac{1}{6}\cdot 42 = 6$` times, i.e. about `$5$` of the `$X$`'s should be `$1$`'s.

What would cause you to reject the idea that the dice are fair?

If we see way more than `$6$` sixes. What would be "way more"? `$7$`? `$8$`? `$17$`?

---

In terms of probabilities: what is the *probability* of observing at least `$10$` sixes *IF* the `$P(X_i = 1) = \frac{1}{6}$`?

* if the probability is small, `$10$` is a lot of sixes
* if the probability is large, `$10$` is a reasonable number of sixes

First, introduce the random variable `$Y =$` number of sixes `$= X_1 + X_2 + ... X_{42}$`. The probability of observing more than `$10$` sixes is `$P(Y \ge 10)$`. To find this, we need the distribution of `$Y$`, which is 
--
 `$\text{Binomial}(42, \pi)$`, where `$\pi$` is the probability of rolling a six.

---

*IF* the dice are fair, `$\pi = \frac{1}{6}$`. So *IF* the dice are fair, the distribution of `$Y$` looks like this:

<center>
<div id="htmlwidget-52c8f3789c441f9a99bb" style="width:600px;height:450px;" class="plotly html-widget"></div>
<script type="application/json" data-for="htmlwidget-52c8f3789c441f9a99bb">{"x":{"data":[{"orientation":"v","width":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"base":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"x":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42],"y":[0.00047248460888655,0.00396887071464701,0.0162723699300528,0.0433929864801407,0.0846163236362744,0.128616811927137,0.158627401376802,0.163159612844711,0.142764661239122,0.107866632936226,0.0711919777379089,0.0414207870475107,0.0214007399745472,0.00987726460363713,0.00409200962150682,0.00152768359202921,0.000515593212309859,0.000157710864941839,4.38085735949553e-05,1.10674291187256e-05,2.54550869730688e-06,5.33344679435728e-07,1.01820347892276e-07,1.7707886589961e-08,2.80374871007713e-09,4.03739814251107e-10,5.27967449405294e-11,6.25739199295162e-12,6.7043485638768e-13,6.47316413063961e-14,5.61007557988773e-15,4.34328431991303e-16,2.98600796994026e-17,1.80970179996377e-18,9.58077423510229e-20,4.37978250747535e-21,1.70324875290706e-22,5.52405000942843e-24,1.4536973709022e-25,2.98194332492761e-27,4.47291498739132e-29,4.36381949989399e-31,2.07800928566384e-33],"text":["x: 0 Probability: 0.00047","x: 1 Probability: 0.00397","x: 2 Probability: 0.01627","x: 3 Probability: 0.04339","x: 4 Probability: 0.08462","x: 5 Probability: 0.12862","x: 6 Probability: 0.15863","x: 7 Probability: 0.16316","x: 8 Probability: 0.14276","x: 9 Probability: 0.10787","x: 10 Probability: 0.07119","x: 11 Probability: 0.04142","x: 12 Probability: 0.02140","x: 13 Probability: 0.00988","x: 14 Probability: 0.00409","x: 15 Probability: 0.00153","x: 16 Probability: 0.00052","x: 17 Probability: 0.00016","x: 18 Probability: 0.00004","x: 19 Probability: 0.00001","x: 20 Probability: 0.00000","x: 21 Probability: 0.00000","x: 22 Probability: 0.00000","x: 23 Probability: 0.00000","x: 24 Probability: 0.00000","x: 25 Probability: 0.00000","x: 26 Probability: 0.00000","x: 27 Probability: 0.00000","x: 28 Probability: 0.00000","x: 29 Probability: 0.00000","x: 30 Probability: 0.00000","x: 31 Probability: 0.00000","x: 32 Probability: 0.00000","x: 33 Probability: 0.00000","x: 34 Probability: 0.00000","x: 35 Probability: 0.00000","x: 36 Probability: 0.00000","x: 37 Probability: 0.00000","x: 38 Probability: 0.00000","x: 39 Probability: 0.00000","x: 40 Probability: 0.00000","x: 41 Probability: 0.00000","x: 42 Probability: 0.00000"],"type":"bar","marker":{"autocolorscale":false,"color":"rgba(89,89,89,1)","line":{"width":1.88976377952756,"color":"transparent"}},"showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":40.8401826484018,"r":7.30593607305936,"b":37.2602739726027,"l":48.9497716894977},"plot_bgcolor":"rgba(255,255,255,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"title":{"text":"Binomial(n = 42, π = 1/6)","font":{"color":"rgba(0,0,0,1)","family":"","size":17.5342465753425},"x":0,"xref":"paper"},"xaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[-2.65,44.65],"tickmode":"array","ticktext":["0","10","20","30","40"],"tickvals":[0,10,20,30,40],"categoryorder":"array","categoryarray":["0","10","20","30","40"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":{"text":"Value of Y","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[0,0.179475574129182],"tickmode":"array","ticktext":["0.00","0.05","0.10","0.15"],"tickvals":[0,0.05,0.1,0.15],"categoryorder":"array","categoryarray":["0.00","0.05","0.10","0.15"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":{"text":"Probability","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":"transparent","line":{"color":"rgba(51,51,51,1)","width":0.66417600664176,"linetype":"solid"},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":false,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.88976377952756,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895}},"hovermode":"closest","barmode":"relative"},"config":{"doubleClick":"reset","showSendToCloud":false},"source":"A","attrs":{"6fc96352bc3a":{"x":{},"y":{},"label":{},"type":"bar"}},"cur_data":"6fc96352bc3a","visdat":{"6fc96352bc3a":["function (y) ","x"]},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>
</center>

---

The probability we want to find is the red area below. We will do this in SAS in just a second. The result is 0.07905.

<center>
<div id="htmlwidget-a695a17bd7a0921c6da9" style="width:600px;height:450px;" class="plotly html-widget"></div>
<script type="application/json" data-for="htmlwidget-a695a17bd7a0921c6da9">{"x":{"data":[{"orientation":"v","width":[1,1,1,1,1,1,1,1,1,1],"base":[0,0,0,0,0,0,0,0,0,0],"x":[0,1,2,3,4,5,6,7,8,9],"y":[0.00047248460888655,0.00396887071464701,0.0162723699300528,0.0433929864801407,0.0846163236362744,0.128616811927137,0.158627401376802,0.163159612844711,0.142764661239122,0.107866632936226],"text":["x: 0 Probability: 0.00047","x: 1 Probability: 0.00397","x: 2 Probability: 0.01627","x: 3 Probability: 0.04339","x: 4 Probability: 0.08462","x: 5 Probability: 0.12862","x: 6 Probability: 0.15863","x: 7 Probability: 0.16316","x: 8 Probability: 0.14276","x: 9 Probability: 0.10787"],"type":"bar","marker":{"autocolorscale":false,"color":"rgba(89,89,89,1)","line":{"width":1.88976377952756,"color":"transparent"}},"name":"Y < 10","legendgroup":"Y < 10","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"orientation":"v","width":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"base":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"x":[10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42],"y":[0.0711919777379089,0.0414207870475107,0.0214007399745472,0.00987726460363713,0.00409200962150682,0.00152768359202921,0.000515593212309859,0.000157710864941839,4.38085735949553e-05,1.10674291187256e-05,2.54550869730688e-06,5.33344679435728e-07,1.01820347892276e-07,1.7707886589961e-08,2.80374871007713e-09,4.03739814251107e-10,5.27967449405294e-11,6.25739199295162e-12,6.7043485638768e-13,6.47316413063961e-14,5.61007557988773e-15,4.34328431991303e-16,2.98600796994026e-17,1.80970179996377e-18,9.58077423510229e-20,4.37978250747535e-21,1.70324875290706e-22,5.52405000942843e-24,1.4536973709022e-25,2.98194332492761e-27,4.47291498739132e-29,4.36381949989399e-31,2.07800928566384e-33],"text":["x: 10 Probability: 0.07119","x: 11 Probability: 0.04142","x: 12 Probability: 0.02140","x: 13 Probability: 0.00988","x: 14 Probability: 0.00409","x: 15 Probability: 0.00153","x: 16 Probability: 0.00052","x: 17 Probability: 0.00016","x: 18 Probability: 0.00004","x: 19 Probability: 0.00001","x: 20 Probability: 0.00000","x: 21 Probability: 0.00000","x: 22 Probability: 0.00000","x: 23 Probability: 0.00000","x: 24 Probability: 0.00000","x: 25 Probability: 0.00000","x: 26 Probability: 0.00000","x: 27 Probability: 0.00000","x: 28 Probability: 0.00000","x: 29 Probability: 0.00000","x: 30 Probability: 0.00000","x: 31 Probability: 0.00000","x: 32 Probability: 0.00000","x: 33 Probability: 0.00000","x: 34 Probability: 0.00000","x: 35 Probability: 0.00000","x: 36 Probability: 0.00000","x: 37 Probability: 0.00000","x: 38 Probability: 0.00000","x: 39 Probability: 0.00000","x: 40 Probability: 0.00000","x: 41 Probability: 0.00000","x: 42 Probability: 0.00000"],"type":"bar","marker":{"autocolorscale":false,"color":"rgba(255,0,0,1)","line":{"width":1.88976377952756,"color":"transparent"}},"name":"Y ≥ 10","legendgroup":"Y ≥ 10","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[9.5,9.5],"y":[0,0.179475574129182],"text":"","type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"rgba(0,0,0,1)","dash":"dash"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":40.8401826484018,"r":7.30593607305936,"b":37.2602739726027,"l":48.9497716894977},"plot_bgcolor":"rgba(255,255,255,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"title":{"text":"Binomial(n = 42, π = 1/6)","font":{"color":"rgba(0,0,0,1)","family":"","size":17.5342465753425},"x":0,"xref":"paper"},"xaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[-2.65,44.65],"tickmode":"array","ticktext":["0","10","20","30","40"],"tickvals":[0,10,20,30,40],"categoryorder":"array","categoryarray":["0","10","20","30","40"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":{"text":"Value of Y","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[0,0.179475574129182],"tickmode":"array","ticktext":["0.00","0.05","0.10","0.15"],"tickvals":[0,0.05,0.1,0.15],"categoryorder":"array","categoryarray":["0.00","0.05","0.10","0.15"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":{"text":"Probability","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":"transparent","line":{"color":"rgba(51,51,51,1)","width":0.66417600664176,"linetype":"solid"},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":false,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.88976377952756,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895}},"hovermode":"closest","barmode":"relative"},"config":{"doubleClick":"reset","showSendToCloud":false},"source":"A","attrs":{"6fc914041bc9":{"fill":{},"x":{},"y":{},"label":{},"type":"bar"},"6fc919bc3363":{"xintercept":{}}},"cur_data":"6fc914041bc9","visdat":{"6fc914041bc9":["function (y) ","x"],"6fc919bc3363":["function (y) ","x"]},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>
</center>

---

This means that *IF* the true probability of rolling a six with these dice is indeed `$\frac{1}{6}$`, the probability of rolling `$10$` or more sixes is `$0.07905$`. This probability is called the *p-value* for the test `$H_0: \pi = 1/6$` when testing against `$H_A: \pi > 1/6$`.

Is this small enough to convince you that the true probability is *NOT* `$\frac{1}{6}$`?

---

.center[
  **In SAS.**
]

---

## Different (but the same) approach

Instead of looking at `$Y$` (number of sixes), look at the proportion of sixes. I.e.

`$$\hat{p} = \frac{Y}{n} = \frac{1}{n}(X_1 + ... + X_{42}) = \frac{1}{n} \sum_{i=1}^{42} X_i$$`

This is an average, so CLT tells us it's normally distributed around the true value of `$\pi$`, and variance `$\frac{\text{Var}(X_i)}{n} = \frac{\pi(1-\pi)}{n}$`.

We reject the idea that the true value of `$\pi$` is `$\frac{1}{6}$` when `$\hat{p}$` is "far from" `$\frac{1}{6}$`.

I.e. when `$\hat{p} - \frac{1}{6}$` is large. When this is large, so is `$Z = \frac{\hat{p}-1/6}{\text{SD}(\hat{p})}$`.

If we pretend `$\pi = \frac{1}{6}$`, then `$Z \sim N(0,1)$`!

If we pretend `$\pi = \frac{1}{6}$`, then `$\text{SD}(\hat{p}) = \sqrt{\text{Var}(\hat{p})} = \sqrt{\frac{1/6 \cdot (1-1/6)}{42}}$`, and so we can actually calculate `$z_{obs}$`, the observed value of `$Z$`.

---

## Different (but the same) approach

So,

`\begin{align*}
\text{p-value} &= P(\hat{p} > p_{obs}) \\
               &= P\left(\frac{\hat{p} - 1/6}{\text{SD}(\hat{p})} > \frac{p_{obs} - 1/6}{\sqrt{1/6 \cdot (1-1/6)/42}} \right)
               &= P(Z > 1.24212)
\end{align*}`

---

So we found our p-value as the area depicted below. Turns out, this is 0.1071.

This strategy can be used every time we the quantity of interest in follows a (at least approximately) normal distribution. Subtract the mean and divide by the standard deviation to get the standard normal. Then find the probability.

---

## Group Comparison

Two groups, 15 observations in each group. Want to test `$H_0: \mu_X = \mu_Y$` against `$H_A: \mu_X \neq \mu_Y$`.

---

Natural to look at `$\bar{X} - \bar{Y}$`.

If `$n$` is "large enough", both of these quantities are normally distributed. From last week, normal minus normal is normal. So `$\bar{X} - \bar{Y}$` is normal!

Next question: exactly what normal would it be *IF* we pretend the null hypothesis is true?

Mean: `$E(\bar{X} - \bar{Y}) = E(\bar{X}) - E(\bar{Y})=$`
--
 `$\mu_X - \mu_Y =$`
--
 `$0$`

Variance (assuming independence): `$\text{Var}(\bar{X} - \bar{Y}) =$`
--
 `$\text{Var}(\bar{X}) + \text{Var}(\bar{Y}) = \frac{\sigma_X^2}{n_X} + \frac{\sigma_Y^2}{n_Y}$`.
 
So `$Z = \frac{\bar{X} - \bar{Y}}{\text{SD}(\bar{X} - \bar{Y})} = \frac{\bar{X} - \bar{Y}}{\sqrt{\text{Var}(\bar{X} - \bar{Y})}} \sim N(0,1)$`.

What's the problem here?
--
 We don't know `$\sigma_X^2$` or `$\sigma_Y^2$`!

---

Luckily, we can simply *estimate* both, and plug them in.

When we do this, the result no longer follows a standard normal distribution...
--
 BUT it follows a particular `$t$`-distribution. How to calculate the exact degrees of freedom is a bit tricky, but it can be done.
 
**For the patient type:**

The degrees of freedom is

`\begin{align*}
  \text{df} &= n_X + n_Y - 2
\end{align*}`

if we are willing to assume `$\sigma_X^2 = \sigma_Y^2$`. In this case we would use `$s_X^2 = s_Y^2 = \frac{1}{n_X + n_Y - 2}\left(\sum_{i=1}^{n_X} (x_i - \bar{x})^2 + \sum_{i=1}^{n_Y} (y_i - \bar{y})^2\right)$`.

---

If you're not willing to assume `$\sigma_X^2 = \sigma_Y^2$`, the degrees of freedom is

`\begin{align*}
  \text{df} &= \frac{\left[\frac{s_X^2}{n_X} + \frac{s_Y^2}{n_Y}\right]^2}{\frac{(s_X^2/n_X)^2}{n_X - 1} + \frac{(s_Y^2/n_Y)^2}{n_Y - 1}},
\end{align*}`

and we would then use the regular sample variances as estimates:

`\begin{align*}
  s_X^2 &= \frac{1}{n_X - 1} \sum_{i=1}^{n_X} (x_i - \bar{x})^2 \\
  &\\
  s_Y^2 &= \frac{1}{n_Y - 1} \sum_{i=1}^{n_Y} (y_i - \bar{y})^2
\end{align*}`

---

**For the not so patient type:**

Don't worry about it, you'll use software to calculate this.

HOWEVER, you will need to know WHEN you can assume equal variance.